feat!: reuse freed memory after a sweep, and drop the analysis - #49

Summary

Implements RFC 0036, Deterministic heap reuse. Under v0.16.0, a use after free was caught only while the freed block sat in a 16 MiB quarantine; once the program allocated enough, the block went to a new owner and the stale pointer corrupted it silently. Now a released heap block is reused only after a sweep finds no pointer to it anywhere in the program. This PR also deletes the advisory ownership analysis, which on untuned code reported nothing worth fixing (62 of 69 findings false) and caught none of the bugs the guards miss.

The RFC's first draft also made spatial safety and use after return deterministic, through bounds from each pointer's provenance and locals moved into arena "frame blocks". That draft was built on this branch and measured at twice v0.16.0's run time (QuickJS 10.8×), so this revision keeps only the cheap parts. The RFC's Alternatives section records the per-mechanism measurement, and those features move to a later RFC that starts from a cost budget.

What a guarded program now gets:

• Use after free and double free trap through any pointer the program still holds, however much it allocated since. The quarantine is no longer a budget a program can exhaust.
• Fortified calls are checked against their object size. A strcpy past an 8-byte field into the next field traps as intra-object-overflow; v0.16.0 dropped the size.
• ASan's contract. Code sees __SANITIZE_ADDRESS__ and __has_feature(address_sanitizer), and the runtime implements ASan's poisoning interface, so code written to be ASan-clean is WeaveC-clean. In the corpus this removed jansson's deliberate hash over-read and SQLite's misuse-test trap, and pools that poison for ASan are now checked (use-after-poison).
• Shared frame space. In frames larger than 512 bytes, locals whose lifetimes never overlap share bytes, as in Clang's own frames. CPython's recursion tests no longer crash.
• Drop-in fixes. Globals of dylibs and bundles get their redzones in the process's shadow, and the strto* / ato* parsers no longer require a terminator they never read.

Measured (darwin-arm64; full table in the RFC's Measured gates)

• Detection (G1–G3). Every determinism case this RFC catches stops, and every twin runs clean. The detection suites stop 105 of 109 bugs (v0.16.0: 104), and no twin stops. Verify mode reports no weavec.proven on any case.

• Untuned programs (G4). CPython 3.13.7, SQLite 3.50.4 and coreutils 9.5 configure as with the reference compiler and pass as it does, with two exceptions:

  • CPython's test_faulthandler, whose test_read_null reads null on purpose and gets WeaveC's report;
  • four coreutils tests that reach gnulib's word-at-a-time rawmemchr over-read, which ASan stops too.

  tr, SQLite's capi3c, test_ast, test_functools and test_regrtest, which failed under v0.16.0, now pass.

• Corpus (G5). Every config builds and passes its suite in trap and report mode with no trap. Injections stop 33 of 39, the same six misses as v0.16.0.

• Run time (G6). The geometric mean is 1.05× v0.16.0's (1.61× the reference compiler). Every benchmark is within 1.25× its v0.16.0 ratio except QuickJS (see below).

• Memory (G7). Peak RSS is within the limit on all 13 benchmarks: brotli 162 MiB (v0.16.0: 441), cmark 365 (473), libgit2 170 (117; limit 181).

• Pauses (G8). The longest sweep with 256 MiB of live heap is 33 ms (limit 150).

• Build (G9). Compile CPU is 0.86× v0.16.0's; no project is above 1.07×.

Not met, carried forward: QuickJS run time (G6). QuickJS runs at 1.31× its v0.16.0 ratio. Under __SANITIZE_ADDRESS__ it sends every object to malloc (8.3 million allocations instead of 76,000), so each object is tracked separately. Built the same way (-D__SANITIZE_ADDRESS__ for v0.16.0), v0.16.0 takes 3.2 s and this tree 2.72 s; with -fno-weavec-asan-compat it is 1.06× v0.16.0's.

Linux (x86-64). Measured by this PR's CI only, because the local Docker daemon does not respond. The Release and Debug + ASan/UBSan jobs pass every test, including the sweep's Linux half (signal-stopped threads, /proc/self/maps roots) and the determinism cases in trap mode. Verify mode has not been run on Linux.

The RFC stays Accepted, because two gates are not fully met: G6 (QuickJS, above) and G1's verify-mode run on Linux. RFC 0029 landed the same way.

Design notes

• The sweep (§3.4, runtime/weavec_sweep.c). It stops every other thread (Darwin: thread_suspend; Linux: a signal whose handler waits), scans registers, stacks, writable mappings and live heap blocks conservatively, and recycles quarantined slots that nothing marked.

  • On Darwin the walk skips mappings with no dirty pages and the kernel's reclaim ring.
  • It scans the writable segments of the loaded shared-cache images, from dyld's callbacks, instead of the cache's whole writable mapping.

  These two changes cut the fixed cost per sweep from 12–25 ms to about 1 ms (amendment 3).

• Pacing (§3.3, amendment 1). A sweep is due at the soft budget max(Q, live / 4), but waits until eight of the last sweep's durations have passed; the hard budget max(2 Q, live / 2) sweeps at once.

• Reuse. Recycling builds free lists in address order (amendment 4). Released slots of 64 KiB or more drop their pages at once, with a fresh mapping on Darwin, because MADV_FREE stayed resident (amendment 5). WEAVEC_RT_STATS=1 now also prints peak live and quarantined bytes.

• Frames (§2, amendment 8). RFC 0035's native frames are back, with StackLifetime-based sharing. Sharing applies only above 512 bytes, so a use after scope in a small frame cannot land in a later local's bytes.

• Removed from the first draft: provenance bounds, escape checks and the escape registry, pointer slots, frame blocks and capture logs, the setjmp hook, the globals table and __weavec_rt_bounds.

• Released huge blocks (amendment 10). A released huge block counts toward the sweep's budget at its size until a sweep unmaps it. Before this, a program that freed only huge blocks never swept: 128 releases of 600 MiB blocks kept 75 GiB of address space reserved, and on Linux used one of the process's limited mappings each.

• Amendments. All ten amendments in the RFC override its body. The rescope is described at the top of the RFC.

Fixed after review

• A sweep that cannot read every root recycles nothing. Before, a Linux sweep that failed to read /proc/self/maps still recycled. A read error part-way through, or Darwin's address-space walk ending early, now also counts as a failed scan.
• Shared memory under /dev is scanned on Linux: /dev/shm, shared anonymous mappings (listed as /dev/zero) and hugetlbfs. Other device mappings are still skipped.
• Stack table. Linux records two stack pointers per stopped thread, so a program with more than 2,048 threads overflowed the sweep's stack table. The table now has room for both. A mapping that holds several stacks is scanned from the lowest stack pointer in it.
• Precompiled headers. A header precompiled by weavec-cc had ASan removed from its recorded options, so every compile that read it with the same flags rejected it. It now keeps them.
• The 512-byte frame-sharing threshold now uses the frame's real unshared size. The old estimate counted each local's alignment as padding, so small frames (three 96-byte locals: 448 bytes, estimated at 544) shared bytes and missed a use after scope.

Size. The library is 7,204 lines (budget 9,000) and the runtime 5,004 (budget 6,000). This PR adds 8.0K lines and deletes 85.5K.

Breaking changes:

• The advisory analysis is deleted, with no aliases:

  • the weavec tool and weavec --whole-program;
  • -fweavec-diagnose, -fweavec-budget=, -fweavec-unit-budget=, -fweavec-dump-analysis and -fweavec-analysis-stats=;
  • every diagnostic id.

  The -W…weavec… flags now reach Clang as unknown warning options.

• weavec.h is version 1.0 and keeps only WEAVEC_UNSAFE. Code that uses another annotation no longer compiles.

• Units are compiled as ASan would compile them (__SANITIZE_ADDRESS__ is defined). -fno-weavec-asan-compat turns this off.

• Optimising glibc builds get -D_FORTIFY_SOURCE=3 unless the command line sets a level.

• Report kinds: intra-object-overflow and use-after-poison are new; buffer-overflow is no longer reported.

• The runtime's entry points and its Darwin dispatch table (version 4) changed, so rebuild every object with this weavec-cc.

• A program that frees heavily can hold more memory between sweeps (up to the hard budget). A block that a stale word still points to stays allocated.

• On Linux the sweep uses SIGPWR (or WEAVEC_RT_SWEEP_SIGNAL).

Test plan

• Unit tests: Core 10, Frontend 14.
• ctest -R '^runtime$': 29 tests, including pacing, library-data-is-scanned, shared-memory-is-scanned, huge-releases-sweep and the sweep-* tests, in release and dev builds.
• lit build/dev/test: 49 tests, including Driver/pch.c and Guards/small-frame-scope.c.
• scripts/run-cases.py: 533 pass and 11 XFAIL in trap mode; verify mode reports no weavec.proven.
• ctest --preset dev: 39 of 39.
• scripts/check-hygiene.py, scripts/check-format.sh, scripts/check-cmake-format.sh, and clang-tidy on the changed C++ files.
• Documentation examples; npm test and npm run build in docs/ (71 pages validated).
• scripts/corpus-gate.py --full (G5), --quick ratchet re-recorded, and --bench against v0.16.0 (G6, G7).
• CPython, SQLite and coreutils by hand (G4); a 256 MiB pause test (G8).
• Linux x86-64 Release and Debug + ASan/UBSan, and clang-tidy (this PR's CI).